Exploration behavior and mental fatigue RUNNING HEAD: EXPLORATION BEHAVIOR AND MENTAL FATIGUE The Impact of Mental Fatigue on Exploration in a Complex Computer Task: Rigidity and Loss of Systematic Strategies

This study investigates the impact of mental fatigue on exploration when performing a complex computer task. Exploration of participants who underwent a fatigue manipulation (N = 36) was compared with a control (non-fatigued) group (N = 32). A distinction was also made between participants with either a low or high level of general computer experience. Results showed fatigued participants to use significantly less systematic exploration and make more errors than non-fatigued participants. Fatigued participants with low experience also showed significantly more rigid behavior than other participants. No differences were found on the number of sub-tasks solved. Compared to low experienced participants, highly experienced participants showed significantly more systematic exploration, less unsystematic trial-and-error, solved more sub-tasks, and made fewer errors (marginally significant p = . 056). Findings were interpreted as the result of reduced task engagement under fatigue and reduced involvement of executive control on behavior. 2 Exploration behavior and mental fatigue The Impact of Mental Fatigue on Exploration in a Complex Computer Task: Rigidity and Loss of Systematic Strategies It is common in modern work places that people have to deal with increasingly complex problems such as managing a big company, finding the fault in a power plant, or learning a new computer program without specific instructions. Although in practice many of these tasks can be executed with routines or through the use of standardized procedures, there are also many complex tasks in which routines or procedures are not directly available or which need ad-hoc actions that go beyond formal procedures. Dealing with such non-routine tasks often requires exploration (Funke, 1991) to gain insight into the task and to find out which actions can accomplish task goals. Exploration can be done in a systematic or unsystematic way (Dörner 1980; Funke, 1991; van der Linden, Sonnentag, Frese, & van Dyck, 2001). Exploring systematically means that people behave in a goal-directed way and reflect on action feedback and on their own behavior (Trudel & Payne, 1995; van der Linden et al., 2001). In contrast, when exploring unsystematically, people often behave in an unstructured way and do not seem to follow a coherent path towards goal attainment (Dörner, 1980; Hollnagel, 1993). Instead, actions are executed impulsively or are guided by external stimuli that tend to capture attention (cf. Hollnagel, 1993). An important factor involved in whether people explore in a systematic versus unsystematic way is their level of task engagement (Dörner, 1980). Task engagement refers to the level of cognitive resources such as attention, allocated to task relevant processes (Dörner, 1980; Trudel & Payne, 1995) and that involve problem solving steps such as goalsetting, hypotheses formation, planning, and feedback evaluation (Dörner, 1980). When many resources are allocated to problem solving, exploration is likely to be goal-directed and systematic. In contrast, when task engagement is low and fewer resources are invested, this tends to manifest itself in the use of unsystematic exploration strategies, which may generally 3 Exploration behavior and mental fatigue be ineffective or inefficient (Hollnagel, 1993; Green & Gilooy, 1990). Although, many factors can influence the level of task engagement, in the current study we focus on one of these factors, namely mental fatigue. Specifically, we measure exploration behavior and assess how systematic versus unsystematic behavior changes under fatigue. As far as we know there is no other study that explicitly investigated exploration under fatigue. Nevertheless, knowing how exploratory behavior may change under fatigue is important because exploration is a substantial part of problem solving in complex tasks (Dörner, 1980; Hollnagel, 1993; Shrager & Klahr, 1986). Mental Fatigue and Exploration Behavior Mental fatigue can be defined as a psychophysiological state resulting from sustained performance on cognitively demanding tasks and coinciding with changes in motivation, information processing, and mood (e.g. Meijman, 2000). One of the main characteristics of mental fatigue is an increased resistance against further effort and a tendency to reduce task engagement (Holding, 1984; Meijman, 2000; Sanders, 1998). Thus, when possible, fatigued people will stop working on effortful tasks and postpone the work until they are no longer fatigued. However, even in situations where one cannot stop working, fatigued people still tend to reduce task engagement (often unintentionally)(Meijman, 2000; Sanders, 1998). Such reduced task engagement will not manifest itself as a complete withdrawal from the task or as a complete break-down of performance. More likely, periods of adequate performance will more frequently be alternated with lapses of task-engagement under fatigue (Sanders, 1998). During such lapses, behavior may not be directed by clear task goals but by more automatic cognitive processes (cf. Monsell & Driver, 2000). With regard to exploration behavior it can be expected that in those lapses, people will not show thoughtful, systematic exploration. Before being to able to study exploration under fatigue it is necessary to establish what are the behavioral manifestations of systematic and unsystematic exploration. Therefore, in the 4 Exploration behavior and mental fatigue following sections we describe three major types of exploration behavior we assess in the current study. With these three types of exploration we do not intent to exhaustively cover all possible forms of exploration behavior but want to capture broad patterns of behavior that people show when working on complex, non-routine tasks (Hollnagel, 1993; Norman, 1991; Trudel & Payne, 1995). We labeled these types of exploration behavior systematic exploration, unsystematic trial-and-error, and rigid behavior. Systematic exploration implies that a person explores a system in a goal-directed, coherent way (Green & Gilooy, 1990; Trudel & Payne, 1995). This means that hypotheses about where to search for and about possible outcomes of actions are generated and that behavior is guided by these hypotheses (Shrager & Klahr, 1986). Moreover, exploring systematically implies to reflect on action feedback and on working methods. Several studies support the importance of goal-directed, reflective behavior for successful exploration (Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Shrager & Klahr, 1986; Trudel & Payne, 1995). For example, Trudel and Payne (1995) analyzed verbalizations of people who explored a digital stopwatch and found that ‘good’ explorers (the ones who learned most about the watch) tended to verbalize discoveries they made earlier, assessed frequently what has been learned so far, and often confirmed or disconfirmed feedback, derived from testing ideas. In general, using systematic exploration involves a thoughtful, reflective approach to the task (Trudel & Payne, 1995, p. 325). As such, it can be argued that systematic exploration involves a relatively high level of engagement. However, as mental fatigue coincides with a reduction in task engagement (e.g. increased lapses of task engagement), it can be expected that the use of systematic exploration will decrease under fatigue (Hypothesis 1). Unsystematic trial-and-error refers to exploration that is unstructured and does not seem to be guided by clear hypotheses, nor is it accompanied with signs of reflection (Hollnagel, 1993; Trudel & Payne, 1995). During unsystematic trial-and-error, people often 5 Exploration behavior and mental fatigue shift from one sub-goal to another, while none or only a few of these sub-goals are wellconsidered. In the literature on problem solving and human-computer interaction (HCI) there are many reports of such behavior even though different labels have been assigned to it, for example ‘vagabonding’ (Dörner, 1980), ‘scrambled mode’ (Hollnagel, 1993), or ‘unsystematic trial-and-error’ (Trudel & Payne, 1995). In general, unsystematic trial-anderror may coincide with a withdrawal of cognitive resources from hypotheses formation, planning, and reflection. Moreover, as the tendency to reduce task engagement increases under fatigue we can expect that the use of unsystematic trial-and-error also increases under fatigue (Hypothesis 2). Rigid behavior is characterized by decreased cognitive flexibility and increased tendency to perseverate. During periods of rigid behavior, actions or ideas are often initially guided by habits or by salient cues that capture attention. Based on such habits or cues people relatively quickly adopt certain action patterns in which they persist even though feedback clearly indicates that this is no longer useful (Dörner, 1980; van der Linden, et al., 2001). Rigid behavior is another specific type of unsystematic behavior often reported in the problem solving or HCI literature. For example, in a study on learning a statistical program through exploration, Green and Gilhooly (1990) found poor learners to show a tendency to repeat methods, to pay less attention to feedback, and to fail to act appropriately on evaluation feedback. A finding that has been replicated in several other studies (e.g. Somsen, van der Molen, Jensen, & van Beek, 2000; Trudel & Payne, 1995; van der Linden, et al. 2001). We expect that a reduction in reflection and in the allocation of attention to action, will lead to an increase in rigid behavior. Thus, we expect an increase of rigid behavior under fatigue (Hypothesis 3). Performance. In order to study exploration, we used a complex computer task (with Microsoft Excel 4.0) in which participants could freely explore. In accordance with the 6 Exploration behavior and mental fatigue literature on exploration, we expected the use of systematic, reflective exploration to lead to better learning of options and procedures (Funke, 1991; Green & Gilooy, 1990; Hollnagel, 1993; Trudel & Payne, 1995; van der Linden, et. al, 2001). Moreover, if more procedures of the program are learned then more sub-tasks can be achieved. Thus, we expected a positive relationship between systematic exploration and performance, in terms of the number of subgoals solved (Hypothesis 4a). Moreover, because systematic exploration involves thoughtful actions we hypothesized this type of exploration to be negatively related to the number of errors (Hypothesis 4b). In contrast, during periods of unsystematic exploration, task engagement is low and so is planning and reflection. This implies that many errors will be made and that relatively little is learned about new procedures or options. Consequently we expected a negative relationship between unsystematic trial-and-error and rigid behavior on the one hand and number of sub-tasks solved on the other hand (Hypothesis 5a). As unsystematic exploration strategies tend to lead to an increase of errors (e.g., rigid behavior leading to ineffective actions) we postulate Hypothesis 5b as a construct validity hypothesis, which states that the use of unsystematic exploration strategies are positively related to the number of errors. With regard to mental fatigue we expected a negative relationship with solved subtasks (Hypothesis 6a) and a positive relationship with errors (Hypothesis 6b). Although such hypotheses make sense, we have to note that there is a surprising number of findings in the literature that do not show clear-cut relationships between fatigue and performance (Hockey, 1997; Holding, 1984; Sanders, 1998). The main reason for this is that people can reallocate resources, thereby forcing themselves to stay engaged in the task despite their fatigue (Hockey, 1997). Nevertheless, we hypothesized a relationship of fatigue with low number of subtasks solved and with a high number of errors, although, we are fully aware that these 7 Exploration behavior and mental fatigue hypotheses are not easily supported in empirical studies on fatigue (Hockey, 1997; Sanders, 1998). Mental Fatigue and General Experience If the allocation of cognitive resources to task relevant processing plays an important role in exploration, we can expect that other factors that influence the availability of resources and the ability to work systematically on a task, moderate relationships between fatigue and exploration. We argue that experience is such a moderator. In complex tasks, demands on cognitive resources during the initial learning phase are high because novices have to guide every step in the problem solving process consciously (Anderson, 1982). However, with growing experience, people develop action procedures that can be executed in a fairly automatic way and that do not require a high level of cognitive resources, such as attention (Anderson1982; Norman & Shallice, 1986). As a result, experienced people work more efficiently and are better able to plan their behavior and to interpret feedback. Thus, compared to low experienced people, experienced people will use more systematic exploration (Hypothesis 7a), and less unsystematic exploration such as trial-and-error and rigid behavior (Hypothesis 7b). Moreover, compared to low experienced people, experienced people will achieve more sub-goals (Hypothesis 8a) and make fewer errors (Hypothesis 8b). Such main effects of experience on exploration can be expected to be even bigger than the effects of mental fatigue on exploration as the effects of fatigue on behavior are often quite subtle (Broadbent, 1979; Hockey, 1997). As experienced people can work on a task more efficiently and without excessive demands on cognitive resources, their behavior and performance may be less susceptible to the influence of sub-optimal states such as mental fatigue (Bainbridge, 1978). Stated differently, lapses in task engagement under fatigue may be less disruptive for experienced people as they efficiently can execute and plan their behavior even when task engagement is 8 Exploration behavior and mental fatigue relatively low. Hence, we expect interactions between the effects of fatigue and the effects of experience on type of exploration behavior. Specifically, the changes in exploration behavior under fatigue will be less strong for experienced people compared to novices (Hypothesis 9). Method Participants Sixty-eight psychology students participated in this study for additional study credits. Participants were randomly assigned to a fatigue or control group. None of the participants had experience with the computer program Excel, which we used in the experimental task. Measures Fatigue. Subjective fatigue was measured with the general activation subscale of the Activation-Deactivation Checklist (AD-ACL, Thayer, 1989, Cronbach’s alpha = .81) and with the Rating Scale Mental Effort (RSME, Zijlstra, 1993, Cronbach’s alpha = .86). Although, the RSME is often used as a single measure of fatigue, it consists of seven 150point answer scales about several fatigue aspects. Two items relate to mental fatigue, two items to physical fatigue, the remaining items measure resistance against further effort, boredom, and visual fatigue. General Computer Experience. We assessed general computer experience with five questions in a 5-point Likert Scale format (Cronbach’s alpha = .83). The questions concerned the frequency of computer use during the last year and experience with a range of computer applications (e.g. Word, Windows, MacOS). In the analyses, participants were assigned to a high or low general experience group, based on a median split procedure. Fatigue manipulation. We used a so-called scheduling task (Taatgen, 1999) on the computer as fatigue manipulation. In this task, participants assigned work time to fictional employees. Adequate planning in this task required strong task engagement as previous 9 Exploration behavior and mental fatigue planning steps had to be kept in mind –taking notes was not allowed-, while participants simultaneous had to think on further planning steps. Computer Tasks. Participants worked on two different tasks on a MacIntosch computer, namely a task with the spreadsheet program Excel (to test our hypotheses) and a task with the graphical program ClarisDraw (for practice purposes). The Clarisdraw Task was introduced to make participants familiar with the exploration method, to allow them to practice thinking aloud (see Procedure section), and to instruct them on how to adequately think aloud. In the Clarisdraw task participants had to reproduce (draw) an example figure that was presented on the screen. The Excel Task was a task with the software program Excel (version 4.0 for Mac). The Excel task was given directly after the manipulation and was used to study exploration. The overall goal in the task consisted of changing the format of a table on the screen according to an example, which was presented in printed form. The task consisted of eight sub-tasks such as moving or adding text, changing text alignment, and adding and coloring table rows. These sub-tasks were not explicitly mentioned in the instructions nor was instructed how to approach the task. Task instruction only mentioned that the table on the screen should look like the example table. For each experimental session, appearance and settings of Excel were standardized so that each participant would start out in the same environment in which only the worksheet, the standard toolbar and the formatting toolbar were visible. Participants could freely explore the program. An exception was the Helpfunction, which we had disabled to reduce behavioral freedom, hence simplifying the coding of behavior. Although disabling Help makes the experiment somewhat artificial compared to real life settings this had no substantial consequences for our study (which looks at exploration of a system without step-by-step instructions). 10 Exploration behavior and mental fatigue Thinking Aloud We used a thinking aloud procedure (Ericsson & Simon, 1993). Participants had to verbalize their thoughts while working on the tasks. Although, verbalizations do not cover all cognitive processes during the tasks (e.g. sometimes people omit information and some processes cannot be verbalized) they have shown to provide useful indications about goals and intentions underlying behavior (Ericsson & Simon, 1993). Procedure Participants were tested individually in sessions that lasted approximately three hours. At the beginning of the session participants filled in questionnaires on general computer experience and level of fatigue (see Measures section). Thereafter they worked on the ClarisDraw task for 15 minutes. Participants had to think aloud and when necessary the experimenter clarified and corrected the thinking aloud procedure during this task. When participants stayed quiet for several seconds the experimenter asked them to “keep on thinking aloud please” (Ericsson & Simon, 1993). After the ClarisDraw task, participants in the fatigue condition continuously worked on the scheduling task for two hours. Participants in the control condition were told to wait for two hours in which they could read magazines or watch videos. After the manipulation participants again filled in fatigue questionnaires. All participants then worked on the Excel task for 15 minutes in which they had to think aloud again. The computer screen of the participants was directly connected to a video recorder that recorded all their actions and verbalizations. Coding of the Data Exploration behavior in the Excel task was coded from the videotapes which contained the participants’ behavior on the computer screen and their verbalizations. Coders were blind to experimental condition and to the participants’ score on the general experience questionnaires. We used three behavioral categories that represented systematic or 11 Exploration behavior and mental fatigue unsystematic exploration (see Introduction), namely, systematic exploration, unsystematic trial-and-error, and rigid behavior (description of categories below). In addition, we had one category for coding non-exploratory behavior. In complex tasks, such as our exploratory computer task, there are often no easy observable beginor end points, therefore, we decided to use fixed time intervals of 20 seconds as coding units. First, coders decided whether a 20second interval contained exploration behavior. If that was the case, the coders assessed whether exploration behavior fell into one of the three exploration categories. From the sixty-eight videotapes analyzed, we randomly assigned twenty videos to each of two independent coders. We used intra-class correlations (ICC) as reported by Shrouth and Fleiss (1979) to assess interrater reliability. ICCs for the categories are reported below (60 < ICC < 75 = good interrater agreement, ICC > .75 = excellent interrater agreement, Ciccetti & Sparrow, 1981). Coding Categories Systematic exploration. Exploration was coded as systematic when the participants tried out functions or ideas in a structured and coherent way. This was the case when participants’ actions either followed from explicit plans (e.g. “I will now try to use this same function for changing this part here..”) or naturally followed from the previous actions (e.g. changing the settings of an option and then try that option again). Moreover, if coders found participants to evaluate what happened then such behavior was coded as systematic exploration too (e.g. “okay..this function serves to..”). ICC for systematic exploration was . 75. Unsystematic trial-and-error. Behavior was coded unsystematic trial-and-error if the participants did not show signs (either in terms of actions or of verbalization) of reflection or of feedback evaluation. For example, when the participants quickly ‘jumped’ from one option 12 Exploration behavior and mental fatigue to another, showing no signs that options were well considered before going to the next one. ICC for this category was .83 Rigid behavior. Behavior was coded as rigid if the participants repeated (more than twice; Trudel & Payne, 1995) the same action sequence that already turned out to be unsuccessful in earlier attempts. Alternatively, behavior was coded rigid when participants continued to come back to the same options despite accumulating evidence that that option did not work. These criteria for coding rigid behavior in the computer task largely resembled perseverative behavior as assessed in traditional psychological tests that are used to diagnose deficits in the regulation of attention (e.g. Heaton, 1985). ICC for this category was .76. Although coding of exploration behavior was done with fixed time intervals, coders took information from long-term goals and task context into account (e.g. actions belonging to a specific hypothesis could be executed over several consecutive 20-second periods). In this way we combined information about goals, intentions, and task context, with a fairly fine-grained analysis of behavior. Non-exploratory behavior. In the Excel task, participants obviously also displayed non-exploratory behavior. For example, from their experience or as a result of exploration, participants discovered procedures in Excel, which they then applied to fulfill the task (e.g. changing the border around several parts of the table). Such application behavior often covered several 20-seconds coding periods. Application behavior was coded under a separate category (ICC = HAVE TO CHECK). In addition, there also was a proportion of behavior that was not exploration, nor application. This behavior was placed under a Residue-category (see Table 2). Performance. In task analysis of the Excel assignment we determined eight sub-tasks that had to be accomplished (see task description). An important performance variable was the number of 13 Exploration behavior and mental fatigue sub-tasks solved within the time given. We also counted number of errors. Errors were defined as actions with negative consequences or actions that had no effect at all. Results Due to the relatively strong statistical power required to detect multivariate effects, we adopted an α of .10 for multivariate tests. For all univariate and post-hoc tests an overall α of .05 was used. Eta-squares (η 2 ) are reported as effect size.